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ABSTRACT 


MIRELE is an interactive tabletop for situated domain learning that helps students learn names of the physical objects 
and their relationships by projecting the names, relationships, and additional information onto the real objects. It is build 
based on camera-projector system and offers a domain-independent fast authoring system. Using the authoring system, 
the teachers can create the learning materials themselves without any programming. MIRELE is designed for 
domain-independence, unobtrusiveness and usability for manual activities, and offers a simple low-cost solution that can 
be utilized very fast in the classroom. 
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1. INTRODUCTION 


An essential part of learning any domain is learning its vocabulary, concepts, and the relationships between 
them. In this paper we focus on the domain of manual-procedural activities (MPAs) (Hodaie, Haladjian 
& Bruegge 2018). Manual-procedural activities involve manipulating real physical objects by following a 
given procedure. Examples of MPAs are numerous, such as repairing a motor, assembling a part, and 
different crafts. In all these activities we a) manipulate physical objects and materials either directly by 
hands, or indirectly using tools; and b) follow steps of a given procedure, either by looking it up in the 
instruction manual, or knowing it by heart, or imitating an instructor. As part of learning a manual-procedural 
activity, a student must learn the names of the objects, materials, and tools, and how they relate to each other. 
Acquisition of this domain knowledge is part of every Vocational Education and Training program (Rauner 
et al. 2012). 

A motivation behind our work is the current refugee crisis facing the European countries. Many of the 
refugees are skilled professionals that could be integrated into the local economies. However, a challenge 
towards incorporating them quickly into the job market is the language barrier. These professionals already 
know their corresponding domain of work, but they don't know the vocabulary in the language of the host 
country. So our system aims to aid them in faster learning of their technical vocabulary. 

In this paper we propose a mixed-reality system based on a camera-projector setup to assist vocabulary 
and concept learning. Our assumption is that a mixed-reality system that involves the real objects and 
provides a situated (Lave & Wenger 1991) and embodied (Abrahamson & Lindgren 2014) learning 
experience will be more effective in learning and retention of the vocabulary and concepts of a domain. 

The system consists of three main components: an interactive tabletop that recognizes and tracks real 
objects and projects names and different additional information onto them; A domain-independent authoring 
system that allows teachers to create learning content without any programming; A simple GUI for creating 
object recognition components. 
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2. RELATED WORK 


The Bloom’s Taxonomy of learning objectives considers acquiring knowledge as the basis of all other 
cognitive processes of learning (Krathwohl 2002). The knowledge itself is further categorized into factual 
knowledge (e.g. terminology), conceptual knowledge (e.g. categories and interrelations between concepts), 
and procedural knowledge (e.g. methods and procedures for doing something). 

There is evidence in the literature that affordances such as embodiment and interactivity provided by 
Augmented Reality (AR) and tangible user interfaces improve learning and knowledge acquisition 
(Schneider et al. 2010, Diegmann et al. 2015). Manches et al. (2009) discuss the perceptual and manipulative 
properties of physical objects and how these properties can support learning for children. Working with 
physical objects and hands-on learning is also an integrated part of engineering disciplines (Carlson 
& Sullivan 1999). Accordingly, we argue that when the domain of learning is the physical world, such as 
tools and objects required for accomplishing a physical task, interacting with the real objects will benefit the 
learning performance. 

Another related area of research for our work is the situated vocabulary learning. Several systems have 
utilized the pedagogical theories of hands-on, authentic, and situated learning (Ozverir & Herrington 2011) in 
the field of vocabulary learning. Ogata et al. (2004) use RFID tags on objects and environment attached 
sensors in order to track user’s environment and present the relevant vocabulary and phrases. Santos et al. 
(2016) created a marker-based mobile vocabulary learning app and showed a slight improvement in learning 
gain as well as a significant positive attitude of the users toward learning. Vazquez et al. (2017) created a 
mixed-reality app using Microsoft Hololens that uses cloud-based object recognition to enable a 
serendipitous language learning experience with dynamic content. 

Our approach is distinguished from related work in two aspects. We offer a fast and domain-independent 
authoring system that allows flexible definition of learning content directly by teachers. Additionally, we 
offer a simple GUI for fast creation of object recognition components that can be used in the system. As a 
result, our system is not limited to hard-coded pre-defined content, and can be used directly by teachers of 
different domains. 


3. MIRELE 


Figure | (left) shows use cases of MIRELE. The system has two actors: the teacher and the student. Teachers 
can define learning content in the Authoring mode. This learning content is presented in Training mode to the 
student. The teacher can also define exercises that are presented to the student in Exercise mode. 


MIRELE 
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Define 
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Figure 1. Use case of MIRELE and the camera-projector setup 
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We considered the following design goals in creating MIRELE: Low-cost, easy to setup, and easy-to-use 
system; and fast and domain-independent authoring. Accordingly, we chose a camera-projector setup, 
because the required equipment can easily be acquired with comparatively low price in the consumer market. 
Furthermore, this setup allows for easy hands-free interaction without the need for bulky and tethered 
head-mounted displays. The same setup is used by the teachers to create the learning content quickly using a 
graphical user interface. Figure | (right) shows our camera- projector setup. 

MIRELE has two modes: Authoring and Training. In Authoring mode, the teacher interacts with the 
camera-projector setup and authoring GUI on a PC to create learning contents. This step includes defining 
objects and their annotations and defining exercises. The annotations can be different multi-media content 
like text, arrows, video, and audio. In Training mode, the student interacts with the tabletop by putting and 
manipulating objects on the table, or selecting the objects and annotations using a tracked selection stick. 
In Training mode, the content defined by the teacher is projected on the tabletop. 

Learning content are presented as AR annotations projected onto the objects on the tabletop. This include 
textual annotations, arrows, circles, checkboxes, images, audio, and video snippets. Each annotation can be 
either attached to an object or placed on a fixed position on the tabletop. Furthermore, the teacher can define 
named relationships between objects. In Training mode, the relationships are shown as straight labeled lines 
between objects. For example, different kinds of pliers can be grouped together using relationship lines. 

The system does not consider the semantics of the content. The teacher defines what are the learning 
content and their semantics. This can be names of the materials, objects, and tools, names of the parts of an 
object, and different relationships between objects and which objects are required for a task. Similarly, 
exercises are defined as membership questions. For example, "Which objects are needed to accomplish task 
X?". The system considers the content merely as different kinds of AR annotations. 

In order to dynamically project annotations onto the physical objects, the table needs to know which 
objects are on the table, where are their positions, and what are their orientations. This is achieved using 
object recognition components. For each set of objects in a domain, that are involved in a learning session, 
the teacher needs first to create and include the required object detectors. An object detector is either based 
on a machine learning model (marker-less object detection) or based on markers. In the case of marker less 
object detection, we offer a simple UI in which the teacher selects objects that need to be detected. The 
program then creates an object detection package that include a trained ML model. Depending on the 
hardware used, this process may take a long time, up to several hours. For marker-based object detection, the 
teacher associates physical objects with the markers, by selecting them in the UI and assigning a marker ID 
to them. In addition to object detection, the system also offers detection of a selection stick, using which the 
student can interact with objects and annotation. For example, selecting an object can trigger an audio 
annotation that names the object. Or selecting a video annotation will start playing it. 


(a) Authoring content (b) Training (c) Authoring exercises 


Figure 2. The system screenshots (see text for descriptions) 


Figure 2 shows the screenshots of the systems in different modes. In Authoring mode (Figure 2a), the 
teacher sees a top-view of the table provided by the camera and graphically interacts with it to define the 
content. The teacher puts the objects on the table and selects them to be tracked by the systems. He then adds 
different kinds of annotations like textual descriptions, arrows, or video snippets attached to the objects or at 
fixed positions on the scene. In the Training mode (Figure 2b), the student puts the objects on the table, the 
system detects the objects and projects the annotations provide by the teacher onto them. The student can 
select objects and annotation, for example selecting a video annotation that demonstrates how to use a tool. 
The system also projects the relationships between objects as labeled straight lines. The teacher defines 
exercises as questions that have objects as answer (Figure 2c). The exercise consists of a question text and a 
set of objects as the correct answer. For example, “Which tool do you use to pull out a nail?” and for the 
correct answer the student should put the “pincer” on the table. It is also possible to define an exercise as a 
multiple-choice question using checkboxes, or as a selection task, in which the student should select the right 
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object. In exercise mode (Figure 2d), the system projects the question on the table. The student answers the 
question by putting objects on the table. The system detects the objects put on the table and gives feedback 
on correctness of the answer. 


4. CONCLUSION AND FUTURE WORK 


In this paper we introduced our ongoing work on MIRELE, an interactive tabletop intended to support 
learning of a task domain in manual-procedural activities. MIRELE offers a simple authoring tool for the 
interactive tabletop, using which the teachers themselves can create learning content, without depending on 
programmers. We believe this low-cost and simple system will lower the entrance barrier for teachers to 
experiment with new possibilities of augmented reality and tangible interaction in the classroom. 
Furthermore, the authoring system together with the domain-independence focus of MIRELE allows rapid 
creation of learning materials for different domains. We believe this opens new opportunities for research on 
the use of augmented reality in the classroom, that is conducted directly by pedagogical experts themselves 
and in which the cost and effort of content creation is significantly reduced. 

We are currently conducting a pilot between-subject user study to evaluate the effectiveness of MIRELE 
for learning the vocabulary of simple task domain consisting of 10 tools from a typical house-hold toolbox in 
German language. Our assumption is that the participant who learn with physical objects will perform better 
in terms of retention of the object names compared to those who use a paper-based learning material. In the 
next step, we plan to conduct usability studies with teachers of a vocational school, in order to evaluate how 
they use MIRELE and get feedback for improving its feature set. 
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